The Movie Quality option on your Nikon D5300 determines how much compression is applied to the video file, which in turn affects the bit rate, or how much data is used to represent 1 second of video, measured in Mbps (megabits per second).
You get just two choices: High and Normal (the default). The High setting results in a higher bit rate, which means better quality and larger files. Choose Normal for a lower bit rate and smaller files.
Because bit rate affects the size of the video file, it also determines the maximum length of the video clip you can create each time you press the movie-record button.
As with the Frame Size/Frame Rate setting, you can select the Movie Quality setting via the Movie Settings option on the Shooting menu. Or you can use the control strip to select both the Frame Size/Frame Rate and Movie Quality setting together.

One last tip about this setting: In the Live View display, a star in the area labeled Movie Quality indicates that the High bit rate is selected. No star means that the option is set to Normal.
The maximum recording time of a single video clip depends on the Frame Size/Frame Rate and Movie Quality options, as outlined here. Note that 60p, 30p, and 24p are the available Frame Size/Frame Rate settings when the Video Mode option is set to NTSC; when that option is set to PAL, available settings are 50, 25, and 24.
Frame Size | FPS | Quality | Maximum Movie Length |
---|---|---|---|
1920 x 1080 | 60, 50 | High, Normal | 10 minutes; 20 minutes |
1920 x 1080 | 30, 25, 24 | High, Normal | 20 minutes; 29 minutes, 59 seconds |
1280 x 720 | 60, 50 | High, Normal | 20 minutes; 29 minutes, 59 seconds |
640 x 424 | 30, 25 | High, Normal | 29 minutes, 59 seconds; 29 minutes, 59 seconds |