ISE and Vivado calculate the Total System Jitter on input and output paths differently.
Vivado analyzes all paths as a register to register path type.
It assumes we have a register outside of the FPGA for the input paths and output paths.
So it calculates as TSJ = (SJ2 + SJ2)1/2 to take both source and destination into account.
This matches the industry standard static timing analysis tools.
ISE analyzes the input and output paths only on the FPGA side.
It only counts the destination for input paths and the source for output paths for Total System Jitter: TSJ = (SJ2)1/2 = SJ.
This is why the TSJ from Vivado is higher than that of ISE and this results in the ISE slack being a bit higher than the Vivado slack on input and output paths.
When virtual clock is used in the set_input_delay/set_output_delay constraints, Vivado only counts for the FPGA side and gives the same Total System Jitter as ISE Timing Analyzer.