Most likely you are up against a limitation of the motherboard, and you should find some documentation on the board before proceeding. Does it have onboard graphics or do you have a separate video card? Most onboard graphics, and some PCI cards, will use a block of RAM for graphics, so something may be interfering with that.
Let's assume your board only has the 3 RAM slots. Some boards don't really allow using all 3, or will lower the RAM clock and/or timing with 3 sticks installed.
All of which requires you to do some reading on your particular board. In any case, re-installing Windows will have no effect. You may find that the best, or only, possible approach is to get 2 sticks of 512MB.
If you want to test the RAM with all 3 sticks installed, download Memtest86 and create a floppy, then run it overnight. If any errors show up, you have bad RAM or an unstable setup.