I managed to build an experimental .shader file which doesn't crash the editor at startup. But it still crashes when I select my shader from the list. For now it's based on the same inputs as the ksPerPixelMultiMap shader, since it is similar to the custom ones I'd like to add.

Crash:

OS-Version: 6.1.7601 (Service Pack 1) 0x300-0x1
g:\acs\kunossim\material.cpp (238): Material::initShaderVars
g:\acs\kunossim\material.cpp (120): Material::setShader

What I'm compiling with:

fxc /T ps_4_0 ksPerPixelMM_pixel.hlsl > ksPerPixelMM_pixel.bin
Then I concatenate it to the ksPerPixelMultiMap.shader in place of the pixel shader, being careful to adjust file length.

pixel shader declarations:

cbuffer cbCamera
{
  float4x4 ksProjection;
  float4x4 ksMVPInverse;
  float4x4 ksCameraPosition;
  float ksNearPlane, ksFarPlane, ksFOV, ksDofFactor;
};
cbuffer cbPerObject
{
  float4x4 ksWorld;
};
cbuffer cbLighting
{
  float4 ksLightDirection;
  float4 ksAmbientColor;
  float4 ksLightColor;
  float4 ksHorizonColor;
  float4 ksZenithColor;
  float ksExposure, ksScreenWidth, ksScreenHeight, ksFogLinear;
  float ksFogBlend;
  float3 ksFogColor;
};
cbuffer cbMaterial
{
  float ksAmbient;
  float ksDiffuse;
  float ksSpecular;
  float ksSpecularEXP;
  float3 ksEmissive;
  float ksAlphaRef;
};
cbuffer cbShadowMaps
{
  float4x4 ksShadowMatrix0;
  float4x4 ksShadowMatrix1;
  float4x4 ksShadowMatrix2;
  float3 bias;
  float textureSize;
};
cbuffer cbCarPaint
{
  float fresnelC, fresnelEXP, fresnelMaxLevel, nmObjectSpace;
  float isAdditive, useDetail, detailUVMultiplier, shadowBiasMult;
  float4 damageZones;
  float dirt, sunSpecular, sunSpecularEXP;
};

sampler samLinear;
sampler samShadow;
sampler samLinearSimple;
Texture2D  txDiffuse;
Texture2D  txNormal;
Texture2D  txMaps;
Texture2D  txDetail;
Texture2D  txShadow0;
Texture2D  txShadow1;
Texture2D  txShadow2;
TextureCube  txCube;

float4 main(
  float4 p0 : SV_POSITION,
  float3 N : NORMAL,
  float3 eye : TEXCOORD0,
  float2 t0 : TEXCOORD1,
  float ext : TEXCOORD2,
  float3 sm0 : TEXCOORD3,
  float3 sm1 : TEXCOORD4,
  float3 sm2 : TEXCOORD5,
  float3 tangent : TEXCOORD6,
  float3 binormal : TEXCOORD7
  ) : SV_TARGET
{
  ...
  return Color;
}

My best guess is maybe cbuffer cbLighting is defined wrong - unlike cbMaterial and cbCarPaint, I couldn't view the dimension of each variable from the editor, or tell whether I had all the variables. The other thing I'm completely guessing at is the main() semantics, and I'm not familiar enough with hlsl to know if the variable names matter there.