You can explore a lot of tea with a very simple setup--a way to heat water, and a gaiwan to both infuse and drink from.
But from your introduction, I'd recommend something just a touch more elaborate:
-an electric kettle that allows you to set the temperature: this will be very helpful when you want to explore teas where temperature makes a huge difference (hotter brings out more nuances faster than cooler water, but also brings out bitterness rapidly, and going straight to boiling water makes most greens undrinkable for me).
This
Bonavita is a good example. I like that this one has presets that enable you to quickly move through the desired temperature range, but also you can set the temperature yourself between those presets. There is a long discussion about similar options
here [and FYI, the Gourmia kettle we discuss at one point is no longer available at Amazon and reviews suggest a lot of quality issues with it, so I can't recommend that anymore.]
There are many tea aficionados who will decry the use of this sort of technology as intefering with the art of tea appreciation, and describe learning to determine the right temperature by listening to the water in your kettle or watching the bubbles, but without a guide to teach you, you're likely to make a lot of errors along the way before you really are expert, and with the kettle, you can bypass that part of the learning curve until or unless you decide you want to do that.
-an inexpensive brewing vesssel: a cheap (but functional) porcelain gaiwan or glass teapot, with a capacity of 6-8 ounces; here in Los Angeles, I'd direct you to a chinese grocery for a basic version that would cost less than $10. A pot with a simple built-in filter is easiest to start, and is a little easier to use than a gaiwan at first. Getting such a cheap and simple gaiwan or pot is difficult online, because they're too cheap to be profitable to mail out one at a time, so you're likely to have to spend more like 15-25$ plus shipping.
Use this first to explore different teas, and see what you like, before getting caught up in discussions about clays and their impact on tea, or the technical and artistic skills of various potters.
($8.99 at my local chinatown tea shop)
Add a cup, and you're all set to start with loose-leaf tea.
I'd strongly recommend avoiding infuser balls/wands, infuser cups/insulated infuser bottles, or kettles with infuser inserts,
I have used a timer when I wanted to be precise about brewing times, but now I'd rather count off the time to myself, because I'm confident I can dilute an overlong infusion back to drinkability and I don't mind the occasional dilute under-infused cup either.
Then you need some tea.
[slipped with some similar advice]