I feel it makes more sense to display related portfolios which are in the same category (instead of previous and next links). You can use portfolio shortcode for displaying the portfolio items that belong to the category of the portfolio item being currently displayed. Hope this works for you.
Another option is to perhaps display a button with link back to the portfolio page for that category instead of next/previous links. Let me know if you need further help. Thanks.
You can hide the next and previous links with the custom CSS –
.single-portfolio .loop-nav { display: none; }