Sealed J-Sword is probably the rarest item in the game, but it isn't the best sword at all (not even when unsealed into Tsumikiri J-Sword, though the special is useful on HUs who build PB too quick due to RAs using specials alot). Best Sword is Dark Flow, with 50 or more hit and Native/Dark or AB/Machine (Machine/Dark is okay for some quests). Dark Flows without 50Hit aren't that good at all if you're attempting to use it well (special).
Heaven Punisher is a really rare handgun (generally, without events or increased rare enemies), but it's usefulness for its rarity isn't worth it.
Psycho Wand is the rarest wand, easily, but it's only useful if you're going to solo in Epsiode 4 (maybe E1 on a FOnewm, and may be necessary for a FOnewearl in E2). Otherwise you're much better using other things.
The ID Cards are extremely rare, but extremely useless too...
GULD. 1/20880 drop, yuuup. [Drop tables will say 1/4545, but that's before DAR). It's really not that good at all, I only find it good for Gush at Falz on a CAST once you make it into a Guld Milla.
I'm really trying to think of things that are really rare, but I don't think there are that many... [Red Ring isn't really rare, before someone says this. 1/36 is an easy rate.]
I also wouldn't necessarily call the PGF weapons rare either, neither PGF itself. It's unobtainable, yes, but when it does come available, it's not that hard to farm...