This is expected behavior since the posts are shown in the order of their creation/published date. The number shown in the load more button is always the remaining posts and does not change the number when a category is chosen. This is the way most grids work since we are trying to filter the posts by JS and not by reloading the page.
The solution in your case would be one of below –
1) Split the grid into two, displayed one below the other – one for each category and remove the filter since the posts will be limited to one category.
2) Or if you want to retain the filter function, you can change the sort order – see ‘Order By’ field in ‘Build Query’ window until you arrive at a good mix of the two category posts when the page loads. You can use the option ‘Menu Order’ in ‘Order By’ field to build a custom ordering of your own by using the plugin – https://wordpress.org/plugins/post-types-order/.
3) Display more posts in one go by changing the ‘Posts per page’ attribute in ‘Build Query’ window to a higher number like 20 or so. This will pull more posts from the other category which currently has only one category post displayed.
Hope this helps.