For 2 \leq m \leq l/2, let G be a simply connected Lie group with Lie
algebra g_0 = so(2m,2l-2m), let
g = k \oplus p be the complexification of the usual Cartan decomposition,
let K be the analytic subgroup with Lie algebra k \cap g_0, and let
U(g) be the universal enveloping algebra of g. This work examines the unitarity
and K spectrum of representations in the "analytic continuation" of
discrete series of G, relating these properties to orbits in the nilpotent
radical of a certain parabolic subalgebra of g.
The roots with
respect to the usual compact Cartan subalgebra are all \pm e_i \pm e_j with
1 \leq i \lt j \leq l. In the usual positive system of roots, the simple root
e_m-e_{m+1} is
noncompact and the other simple roots are compact. Let q = l \oplus u be
the parabolic subalgebra of g for which e_m-e_{m+1} contributes to u
and the other simple roots contribute to l,
let L be the analytic subgroup of G with Lie algebra l \cap g_0,
let L^ C = Int_g(l),
let 2\delta(u) be the sum of the roots contributing to u, and let
\bar{q}=l \oplus \bar{u} be the parabolic subalgebra opposite to
q.
The members of u \cap p are nilpotent members of g. The group
L^C acts on
u \cap p with finitely many orbits, and the topological closure of each
orbit is an irreducible algebraic variety. If Y is one of these varieties,
let R(Y) be the dual coordinate ring of Y; this is a quotient of the
algebra of symmetric tensors on u \cap p that carries a fully reducible
representation of L^C.
For an integer s, let \lambda_s = \sum_{k=1}^m (-l+s/2)e_k.
Then \lambda_s defines a one-dimensional (l,L) module C_{\lambda_s}.
Extend this to a (\bar{q},L) module by having \bar{u} act by 0, and
define
N(\lambda_s+2\delta(u)) = U(g) \otimes_{\bar{q}} C_{\lambda_s+2\delta(u)}.
Let
N'(\lambda_s+2\delta(u)) be the unique irreducible quotient of
N(\lambda_s+2\delta(u)). The representations under study are
\pi_s = \Pi_S(N(\lambda_s+2\delta(u))) and
\pi'_s=\Pi_S(N'(\lambda_s+2\delta(u))), where S = \dim(u \cap k) and
\Pi_S is the S th derived Bernstein functor.
For s \gt 2l-2, it is known that \pi_s = \pi'_s and that \pi'_s is in the
discrete series. Enright, Parthsarathy, Wallach, and Wolf showed for m \leq
s \leq 2l-2 that
\pi_s = \pi'_s and that \pi'_s is still unitary. The present paper shows
that \pi'_s is unitary for 0 \leq s \leq m-1 even though \pi_s \neq \pi'_s,
and it relates the K spectrum of the representations \pi'_s to the
representation of L^C on a suitable R(Y) with Y depending on s. Use
of a branching formula of D. E. Littlewood allows one to obtain an explicit
multiplicity formula for each
K type in \pi'_s. The chief tools involved are an idea of B. Gross and
Wallach, a geometric interpretation of Littlewood's theorem, and some
estimates of norms.
It is shown further that the natural invariant Hermitian form on \pi'_s does
not make \pi'_s unitary for s \lt 0 and that the K spectrum of \pi'_s in
these cases is not related in the above way to the representation of L^C
on any R(Y).
A final section of the paper treats in similar fashion
the simply connected Lie group with Lie algebra
g_0=so(2m,2l-2n+1),
2 \leq m \leq l/2.