From what you wrote...
The transformer box originally had a photoelectric sensor attached to two wires that connect to the box at terminals marked for "photocell." ... When I replaced the photocell I used a photo electric switch with three wires marked as described. When I attached the white and black wires to the inputs it worked.
What happened here is that "3-wire" device that you bought as the replacement for the original "2-wire" device was not at all the same device as the latter. Yet, it mysteriously worked! So, how does one explain that?
Before I get to that, let's cover some background info first.
The original "2-wire" device is a simple common CdS (Cadmium Sulfide) photocell, which is also called a photoresistor. When in darkness, it appears as an open circuit. When light is shone on it, the photons knock some electrons "loose", and the device starts conducting some current. The photocell appears as a resistor, whose resistance diminishes as the light intensity increases.
...I guess I could hook the photocell up to a multi-tester and figure out whether it generates a voltage in the sun -- at least that will tell me something.
You would test it with an ohmmeter (not a voltmeter), and verify that its resistance goes down with exposure to light.
Following is a picture of this common photocell. Note that this type of photocell does not generate electricity as the solar cell in PV arrays.
Your low-voltage lighting transformer incorporates a small electronic circuit that senses the resistance of this photocell, and when the current through the photocell drops to a certain threshold, turns on a relay (what you called a "solenoid"). This relay is the switch that handles the high current that the low-voltage bulbs require.
The "3-wire" device also uses the photocell. However, it incorporates a kind of transistor called a triac. The current through the photocell controls the triac, which itself switches the AC current through the load. This "3-wire" photo switch should be wired according to the diagram that timeasterday has posted.
So, how could this "3-wire" high-voltage switch work as a substitute for the original simpler "2-wire" photocell?
That brings us back to the original question that I posed earlier.
For the answer, let's look at a typical diagram of the "3-wire" device.
Note the top "load" wire is connected to one of the "120 VAC" lines. That is your white wire. The other "120 VAC" is the black wire (going to one end of the photocell and also to pin 1 of the triac), and the remaining "load" wire is the red wire (the one going to pin 2 of the triac).
We can see that by using only the white and black wires, you are wiring up to the photocell, but with a 47K resistor in series with the photoresistor. The triac is not used at all in this manner.
... I left the the red wire dangling. I probably have a device that does a couple of things and I am just tapping one of them...
What you surmised turns out to be correct! You only used the photocell of the replacement photo switch. Obviously, the circuit inside the low-voltage transformer is sensitive enough to work, despite the extra series 47K resistor. Note that in full day-light, the typical photocell resistance may drop to a few K Ohms, which is a lot lower than that 47K value.
By the way, Radio Shack has a pack of 5 photoresistors for $3.19. However, they are the "naked" type, and may get ruined with exposure to the weather. Also, the circuit inside the LV transformer isolates the photocell from the high-voltage AC, so it should be safe to handle the photocell. On the other hand, one should only open up the "3-wire" device to mess with its innards with extra care if the circuit is live.
....
