Longterm investing in 1 or more funds

does it make sense to sell all the VT/VTI to invest in the larger fund?

No - the price of the fund is largely irrelevant. Most (if not all) brokers support fractional shares of funds so you can buy 1 unit of a $10 ETF or 0.1 units of a $100 ETF. The value is the same.

The NAV (total size) is also largely irrelevant. a small S&P 500 tracking fund should have the same performance as a gargantuan S&P 500 tracking fund - there might be a very slight difference in fees as larger funds have economies of scale, but performance wise they should be the same.

It's the relative change that compounds, so whether a fund's price goes from 10 to 12, or 100 to 120 is irrelevant. Both are a 20% gain.

Will I be losing a significant amount of gains in the long run by not bucketing all my assets into the same fund?

No - compounding is not affected by splitting into two different funds. If both increase by 20%, your overall increase is 20% as well. There's no benefit or harm in splitting or combining finds.

The main reason for splitting funds is to diversify. That reduces the risk (volatility) of your portfolio by spreading gains and losses across multiple funds. Diverse funds go up and down by different amounts, so the overall effect is muted compared to having one fund.

Note that diversification only happens if you have funds that are not highly correlated, for example if they target different market segments. VTI is a "whole market" fund, while SWPPX tracks the largest 500 companies, so the vast majority of the performance of VTI is driven by the components of SWPPX, and they will be very highly correlated, reducing the benefit of diversification.

VT is a "all world" fund which will have some diversification benefit when paired with SWPPX ot VTI - combining them will just give you larger exposure to US stocks than VT alone.

I would be interested in knowing if the answer changes depending on whether we use a taxable vs non-taxable brokerage account.

This can affect whether you invest in tax-advantaged securities like municipal bonds funds, or high dividend stocks where the dividend has tax advantages. If you are in a tax-deferred account and reinvest dividends, there's no real benefit to owning high dividend funds.