Why doesn't PmWiki use hierarchical / nested groups?

It essentially comes down to figuring out how to handle
page links between nested groups; if someone can figure out
an obvious, intuitive way for authors to
do that, then nested groups become plausible. See DesignNotes
and PmWiki:HierarchicalGroups.

Why don't PmWiki's scripts have a closing ?> tag?

All of PmWiki's scripts now omit the closing ?> tag. The tag is not
required, and it avoids problems with unnoticed spaces or blank
lines at the end of the file. Also, some file transfer protocols
may change the newline character(s) in the file, which can also
cause problems. See also the Instruction separation page in the PHP manual.

Does PmWiki support WYSIWYG editing (or something like the FCKEditor)?

Short answer: PmWiki provides GUI buttons in a toolbar for common markups, but otherwise does not have WYSIWYG editing. For the reasons why, see PmWiki:WYSIWYG.