Refreshing a lightning datatable after updating selected records from modal(lightning:overlayLibrary) window can be achieved using events. For that we have to pass the event name as attribute during dynamic component creation in js controller. so that we can register and fire an event from overlay component where it can be handled in lightning datatable component, so that we can call the server method to get recently updated records to get it displayed in UI.
Lightning solution Explained :-
Parent component – which displays set of related contacts for an account using Lightning:datatable base component. In Js controller i utilised the dynamic component creation for firing the lightning:overlayLibrary which gonna display the modal popup window where user can able to select the account from Lookup input field(LDS utilised for showing the input field as lookup). Also this example code contains lightning datatable column sorting and record id also embedded in contact record names for easy record navigation.
Git Repo link : https://github.com/dineshdk/Lightning-datatable