I would use all of the genes that were analyzed in your experiment: these are those on which you performed the differential expression analyses (you maybe previously filtered them to remove low-expression genes, etc.), because (under assumption of independence) those are the ones which had a chance of appearing as DEGs.
Regarding filtering out genes which do not map to GO terms, you can control for this at the GO enrichment step: the
goseq function can control this with its
use_genes_without_cat argument. And by default (most recent version) these genes are ignored in the enrichment testing.